Video synthesis system, video synthesis device, and video synthesis method

ABSTRACT

A video display system performs a video conversion process on a video of a camera mounted on a vehicle and displays a resulting video, and includes a plurality of cameras, a detecting unit that detects an object of interest around the vehicle based on information or the like acquired through the plurality of cameras, other sensors, or a network, a transforming/synthesizing unit that transforms and synthesizes the videos photographed by the plurality of cameras using a shape of a virtual projection plane, a virtual viewpoint, and a synthesis method which are decided according to position information of the object of interest detected by the detecting unit, and a display unit that displays the video that is transformed and synthesized by the transforming/synthesizing unit.

TECHNICAL FIELD

The present invention relates to a video synthesis device and a videodisplay device which are capable of transforming and synthesizing videosphotographed by one or more cameras and causing a resulting video to bedisplayed on a display device.

BACKGROUND ART

A system of photographing an area around a vehicle through a pluralityof cameras installed in the vehicle and displaying a video in thevehicle so that a driver can check the area around the vehicle whiledriving the vehicle has been already proposed, and such a known systemalso includes a technique of displaying a video in which the area aroundthe vehicle is viewed from a virtual viewpoint by appropriatelytransforming and synthesizing the videos photographed by a plurality ofcameras installed in the vehicle.

In such a video display system, a technique of arranging a projectionplane on a virtual space, mapping a camera video, creating a videoviewed from a virtual viewpoint in the space, and displaying the videois known. In this case, it is common that, when a difference occurs in ashape between a subject photographed by the camera and the virtualprojection plane, a video that is transformed and synthesized is viewedto be distorted due to the difference. For example, when the subjectphotographed by the camera is a three-dimensional (3D) object such as aperson or a guardrail, and the virtual projection plane is a plane thatis equal in height to the ground surface and parallel to the groundsurface, the 3D object is distorted to be an elongated video in a videothat is transformed and synthesized, and thus the user is unable tounderstand a situation. In order to solve such a problem, varioustechniques regarding a method of generating a video that is transformedand synthesized have been disclosed in the past.

For example, Patent Document 1 discloses a technique of changing a shapeof a virtual projection plane and a position and an angle of a virtualviewpoint according to a state of a vehicle.

CITATION LIST Patent Document

Patent Document 1: JP 2009-171537 A

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

In other words, in Patent Document 1, a technique of changing a displayvideo together with a driving scene by dynamically changing the shape ofthe virtual projection plane according to the state of the vehicle or anobject around the vehicle is disclosed, but a specific countermeasurefor reducing distortion of a specific subject when the subject isdesired to be noticeably displayed is not disclosed, and thus therestill remains a problem in that the subject is displayed to be distortedin the video that is transformed and synthesized.

In this regard, the present invention was made to solve the problems inthe related arts, and it is an object of the present invention toprovide a video display system, a video synthesis device, and a videosynthesis method, which are capable of reducing, even when a specificsubject is desired to be noticeably displayed, distortion of the subjectand creating an image for displaying an object of interest morenaturally.

Solutions to Problems

In order to achieve the above object, according to the presentinvention, as an example, as stated in claims set forth below, first,provided is a video display system that includes an imaging unitincluding a plurality of cameras, a video synthesis device thatsynthesizes a plurality of videos input from the plurality of cameras ofthe imaging unit, and a video display unit that displays a videosynthesized by the video synthesis device, the video synthesis deviceincludes a detecting unit that detects an object of interest based onthe video input from the imaging unit, a transforming/synthesizing unitthat transforms and synthesizes a plurality of videos photographed bythe plurality of cameras of the imaging unit, and an operationprocessing device that controls at least operations of the detectingunit and the transforming/synthesizing unit, and the operationprocessing device transforms and synthesizes the videos photographed bythe plurality of cameras so that distortion of the object of interest isreduced using a virtual projection plane, a virtual viewpoint, and asynthesis method decided according to information related to the objectof interest detected by the detecting unit through thetransforming/synthesizing unit, and causes a resulting video to bedisplayed on the video display unit.

Further, in order to achieve the above object, according to the presentinvention, provided is a video synthesis device that receives aplurality of videos from a plurality of cameras, transforms andsynthesizes the videos, and causes a resulting video to be displayed ona video display unit, and includes a detecting unit that detects anobject of interest based on the videos input from the imaging unit, atransforming/synthesizing unit that transforms and synthesizes theplurality of videos photographed by the plurality of cameras of theimaging unit, and an operation processing device that controls at leastoperations of the detecting unit and the transforming/synthesizing unit,and the operation processing device transforms and synthesizes thevideos photographed by the plurality of cameras so that distortion ofthe object of interest is reduced using a virtual projection plane, avirtual viewpoint, and a synthesis method decided according toinformation related to the object of interest detected by the detectingunit through the transforming/synthesizing unit, and causes a resultingvideo to be displayed on the video display unit.

Further, in order to achieve the above object, according to the presentinvention, provided is a video synthesis method of receiving a pluralityof videos from a plurality of cameras, synthesizing the videos, andcauses a synthesized video to be displayed on a video display unit, andincludes transforming and synthesizing the videos photographed by theplurality of cameras so that distortion of the object of interest isreduced using a virtual projection plane, a virtual viewpoint, and asynthesis method decided according to information related to the objectof interest detected by the detecting unit and causing a resulting videoto be displayed on the video display unit.

Effects of the Invention

According to the present invention, it is an effect in that it ispossible to provide a video display system, a video synthesis device,and a video synthesis method, which are capable of creating an image fordisplaying an object of interest more naturally.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an example of configurations of avideo synthesis device and a video display system according to anembodiment of the present invention.

FIGS. 2A and 2B are a left side view and a plane view illustratingexemplary installation of an in-vehicle camera in the video displaysystem, and FIG. 2C is a diagram illustrating an angle of view of the-vehicle camera and an overlapping region of a photographing range.

FIG. 3 is a diagram for describing a process of performing coordinateconversion from pixels of a photographing camera video into pixelsviewed from a virtual viewpoint in the video display system.

FIGS. 4A and 4B are a left side view and a plane view of a first layoutfor setting a virtual projection plane and a virtual viewpoint in thevideo display system.

FIG. 5 is a diagram illustrating a video viewed from a virtual viewpointwhen a setting of FIGS. 4A and 4B is performed in a photographingcamera.

FIGS. 6A and 6B are a left side view and a plane view of a second layoutfor setting a virtual projection plane and a virtual viewpoint in thevideo display system.

FIG. 7 is a diagram illustrating a video viewed from a virtual viewpointwhen a setting of FIGS. 6A and 6B is performed in a photographingcamera.

FIG. 8 is a diagram illustrating a settable range of a virtual viewpointin a condition of FIG. 6B.

FIG. 9 is a diagram illustrating another setting of a virtual viewpointin an object of interest in FIG. 6B.

FIG. 10 is a diagram illustrating a video viewed from a virtualviewpoint at the time of setting of FIG. 9.

FIGS. 11A and 11B are diagrams for describing a setting of a method ofsynthesizing a video at the time of setting of FIG. 6B and FIG. 9.

FIG. 12 is a flowchart illustrating a process of deciding a method ofdeciding a virtual projection plane and a virtual viewpoint and asynthesis method of a camera video according to a position of an objectof interest in the video display system.

FIG. 13 is a diagram illustrating an example in which an overlappingregion of a camera video is indicated by angle information.

FIG. 14 is a diagram illustrating an example of a change in blendingrates of two camera videos at an angle illustrated in FIG. 13.

FIG. 15 is a diagram illustrating another example of a change inblending rates of two camera videos at an angle illustrated in FIG. 13.

FIG. 16 is a block diagram illustrating configurations of a videosynthesis device and a video display system (including a plurality ofmonitors) according to another embodiment of the present invention.

FIG. 17 is a block diagram illustrating configurations of a videosynthesis device and a video display system (including a plurality oflightings) according to another embodiment of the present invention.

FIG. 18 is a block diagram illustrating configurations of a videosynthesis device and a video display system (including a plurality ofspeakers) according to another embodiment of the present invention.

MODE FOR CARRYING OUT THE INVENTION

Hereinafter, exemplary embodiments of the present invention will bedescribed in detail with reference to the appended drawings.

FIG. 1 is a block diagram illustrating an example of an overallconfiguration of a video display system according to the presentembodiment. Referring to FIG. 1, the video display system basicallyincludes a video synthesis device 100, an imaging unit 110 including aplurality of cameras (n cameras) that image a subject, and a monitor 120that displays an image. In this configuration, a configuration in whichvideos photographed by a plurality of cameras constituting the imagingunit 110 is transformed and synthesized by the video synthesis device100 and then output to the monitor 120 is similar to a common videodisplay system.

Here, first, for a plurality of cameras constituting the imaging unit110, in the present embodiment, for example, four cameras 110-1 to 110-4are installed on front and rear sides and left and right sides of anautomobile 300 as illustrated in FIGS. 2A to 2C, and an example in whichthe video display system according to the present invention is appliedto a vehicle will be described below. However, the present invention isnot limited thereto and may be applied to any other system that displaysphotographed images of a plurality of monitoring cameras. In thedrawings, reference numeral 310 denotes the ground surface.

Referring back to FIG. 1, the video synthesis device 100 includes adecoding unit 101, a central control unit (CPU) 102, a memory 103, adetecting unit 104, a transforming/synthesizing unit 105, an encodingunit 106, and a bus 107. Hereinafter, processes in the respective unitswill be described.

The decoding unit 101 converts video signals input from a plurality ofcameras 110-1 to 110-4 of the imaging unit 110 into informationnecessary for transformation and synthesis of an image such asquantization data.

The CPU 102 controls an operation of the device in general, and decidesa method of synthesizing a shape of a virtual projection plane and avirtual viewpoint necessary for an image conversion process based on aposition of an object of interest detected or determined by thedetecting unit 104 which will be described later.

Each unit connected to the bus 107 writes or reads information necessaryfor video synthesis in or from the memory 103. Example of informationstored in the memory 103 include image data that is input from thedecoding unit 101 and temporarily stored before image conversion,position information of an object of interest detected by the detectingunit 104, and image data that is converted by thetransforming/synthesizing unit 105 and temporarily stored. There arecases in which the transforming/synthesizing unit 105 or the detectingunit 104 is implemented by software. In this case, the function of thetransforming/synthesizing unit 105 or the detecting unit 104 may beimplemented such that the CPU 102 reads a program stored in a storageunit (not illustrated) out to the memory 103, and performs an operationaccording to the program.

The detecting unit 104 has a function of detecting or determining aposition of a target or a certain range which is to be noticeablydisplayed among all videos photographed by the cameras 110-1 to 110-4 ofthe imaging unit 110. Here, the “object of interest” is assumed toinclude a certain range considered to be risky in terms of driving suchas a right front area when taking a right turn in an intersection or abehind area when going backward in addition to a 3D object such as apedestrian, a vehicle that is traveling excluding a subject vehicle, aguardrail, or a building.

As a method of detecting or determining the object of interest, forexample, in addition to the detecting unit 104, a feature quantitydetection processing unit that detects a feature quantity from a cameravideo input through the decoding unit 101 and detecting a relativeposition to a camera position may be provided, or a distance measuringunit that detects a rough direction or distance from its own vehiclethrough one or more sonars or a laser distance meter may be usedtogether. Alternatively, a communication processing unit that acquiresinformation of a global positioning system (GPS) or an intelligenttransport system (ITS), or the like may be provided, and the detectingunit may determine a relative position between its own vehicle and anobject of interest based on the information acquired by thecommunication processing unit. Further, when a plurality of objects aredetected around a vehicle, a controller area network (CAN) informationacquiring unit that acquires vehicle information such as a travelingdirection, a vehicle speed, or a steering angle of a vehicle obtainedthrough a CAN (a communication unit) may be provided, a direction inwhich a vehicle travels may be estimated based on the information, andan object in the direction may be determined to be an object having ahigh collision possibility and determined to be an object of interest bythe detecting unit 104. Further, when a plurality of objects aredetected around a vehicle, an object closest to a vehicle may bedetermined to be an object of interest, and a video may be generated.Here, information such as a relative position of an object of intereston a plane view with respect to its own vehicle or a mounting positionof each camera may be used as information to be detected or determined,and particularly, more detailed information such as a height or a 3Dshape of a 3D object may be used in order to improve an image conversionaccuracy of an object of interest. Through this method, it is possibleto appropriately give a presentation to the user focusing on an objectaround a vehicle or an object serving as an obstacle in traveling of avehicle. The detecting unit 104, the feature quantity detectionprocessing unit, distance measuring unit, the communication processingunit, the CAN information acquiring unit, and the like have a commonaspect in which they acquire information related to an object ofinterest and can be included in the video synthesis device 100 as oneprocessing unit (an object-of-interest information acquiring unit)having one or more functions of these processing units.

Further, when a device capable of acquiring the information related tothe object of interest is included in a vehicle separately from thevideo synthesis device 100, the video synthesis device 100 may includean interface that performs communication with the device and acquiresthe information related to the object of interest, and in this case, thesame function can be implemented although the detecting unit 104 is notprovided.

The transforming/synthesizing unit 105 performs a transformation processand a synthesis process on images input from the cameras of the imagingunit 110 via the decoding unit 101. The transformation process and thesynthesis process will be described later with reference to FIG. 3. Atthis time, the images photographed by the cameras 110-1 to 110-4 of theimaging unit 110 are projected onto an object plane generated in a 3Dspace as the virtual projection plane and converted into images viewedfrom the virtual viewpoint, and the images of the n cameras are furthersynthesized. As described above, when the images of the cameras 110-1 to110-4 are transformed by the transforming/synthesizing unit 105, thetransformed images have a certain overlapping region, and a blendsynthesis process is performed on the overlapping region through ablending which will be described later.

The encoding unit 106 converts an image transformed and synthesized bythe transforming/synthesizing unit 105 into information of a formatnecessary for video display, and outputs the information to the monitor120.

As a result, the video synthesis device 100 can create a quasi-3D spacebased on two-dimensional (2D) camera images. At this time, thetransforming/synthesizing unit 105 projects the image onto theprojection plane virtually arranged on the 3D space, and calculatesimages viewed from the viewpoint virtually arranged on the 3D space. Inthis technique, the images are mapped to the virtual 3D space, and 2Dimages are generated as a video viewed from the virtual viewpoint, butthe present invention is not limited thereto, and thetransforming/synthesizing unit 105 may perform the mapping to the 3Dspace by itself, or an equivalent process to the mapping to the 3D spacemay be performed in mapping to a 2D space in a quasi manner.

As information that is written in or read from the memory 103, inaddition to the above-described information, for example, informationsuch as mounting information of the cameras 110-1 to 110-4 that aremounted in a vehicle in a 3D space coordinate system (for example,coordinate information and angle information), a distortion coefficientof a lens, a focal distance, or an effective pixel size of an imagingsensor may be included, or information necessary for implementing themapping to the 3D space through the mapping to the 2D space in the quasimanner in the transforming/synthesizing unit 105 may be included.Regarding the distortion coefficient of the lens, a coefficientindicating a degree of distortion according to a distance from a centerof a camera image, information indicating a relation between an incidentangle of a lens and a length of a subject, and the like may be included.Particularly, in the present embodiment, as will be described later,there are cases in which a fish-eye lens is used for the cameras 110-1to 110-4, and in the fish-eye lens, distortion in a peripheral portionis larger than that in a central portion, and thus the distortioncoefficient is necessary when it is corrected. This information is usedfor image transformation or image synthesis in thetransforming/synthesizing unit 105.

The memory 103 also stores information related to the virtual projectionplane, the virtual viewpoint, and the synthesis method used by thetransforming/synthesizing unit 105. As the information of the virtualviewpoint, coordinate information (Xv, Yv, Zv) and angle information(αv, (βv, γv) with respect to each axis are stored. As the informationof the virtual projection plane, for example, central coordinates and aradius of a spherical object are stored as a combination of thespherical object and a plane that is parallel to the ground surface andhas the same height as the ground surface. This information may be amore complicated shape or a combination of a plurality of shapes or maybe information described by (Xt, Yt, Zt) used as information of a 3Dobject that is commonly known. As the information related to thesynthesis method, information related to α blending which will bedescribed later is stored as the synthesis method of the overlappingregion.

Next, an example of a process of performing coordinate conversion frompixels of a photographing camera image into pixels of an image viewedfrom the virtual viewpoint will be described below with reference toFIG. 3.

In FIG. 3, a pixel of an image 201 photographed by an actual in-vehiclecamera (here, representatively indicated by reference numeral 110) isindicated by a point Pr (xr, yr, zr) in a camera coordinate system (Xr,Yr, Zr) for the camera 110. Here, in the camera coordinate system, forexample, a depth direction with respect to the camera 110 is indicatedby Zr, a horizontal direction of a photographed image is indicated byXr, and a vertical direction is indicated by Yr. The point Prcorresponds to a point Pw (xw, yw, zw) in a world coordinate system (Xw,Yw, Zw) that is used in common in a space. A pixel of an image 211 whenthe point Pw in the world coordinate system is photographed from avirtual viewpoint 210 arranged in a certain virtual position correspondsto the point Pv (xv, yv, zv) in the camera coordinate system (Xv, Yv,Zv) with respect to the virtual viewpoint 210. In other words, thefollowing coordinate conversion is performed to generate an image viewedfrom a certain virtual position.

A relation between the point Pr in the camera coordinate system and thepoint Pw in the world coordinate system for the actual camera 110 isindicated by the following Formula (1) using a 4×4 perspectiveprojection transformation matrix Mr.

$\begin{matrix}{\begin{bmatrix}x_{r} \\y_{r} \\z_{r} \\1\end{bmatrix} = {M_{r}\begin{bmatrix}x_{w} \\y_{w} \\z_{w} \\1\end{bmatrix}}} & \left\lbrack {{Math}.\mspace{14mu} 1} \right\rbrack\end{matrix}$

A relation between the point Pv in the camera coordinate system and thepoint Pw in the world coordinate system for the virtual viewpoint 210 isindicated by the following Formula (2) using a perspective projectiontransformation matrix Mv.

$\begin{matrix}{\begin{bmatrix}x_{v} \\y_{v} \\z_{v} \\1\end{bmatrix} = {M_{r}\begin{bmatrix}x_{w} \\y_{w} \\z_{w} \\1\end{bmatrix}}} & \left\lbrack {{Math}.\mspace{14mu} 2} \right\rbrack\end{matrix}$

Here, Mp and My include a 3×3 rotating matrix R and a 3×1 translationmatrix T as in the following Formula (3).

$\begin{matrix}{{M_{r} = \begin{bmatrix}R & T \\0 & 1\end{bmatrix}},{M_{v} = \begin{bmatrix}R & T \\0 & 1\end{bmatrix}}} & \left\lbrack {{Math}.\mspace{14mu} 3} \right\rbrack\end{matrix}$

Using Formulas (1) and (2), a relation of the point Pr in an actualcamera coordinate system and the point Pv in the virtual viewpointcoordinates can be obtained by the following Formula (4), that is, apixel value of a virtual viewpoint that is desired to be generated canbe obtained by a calculation based on a pixel value photographed by anactual camera.

$\begin{matrix}{\begin{bmatrix}x_{v} \\y_{v} \\z_{v} \\1\end{bmatrix} = {M_{v}{M_{r}^{- 1}\begin{bmatrix}x_{r} \\y_{r} \\z_{r} \\1\end{bmatrix}}}} & \left\lbrack {{Math}.\mspace{14mu} 4} \right\rbrack\end{matrix}$

A 4×4 matrix obtained by multiplying My by an inverse matrix of Mr usinginformation such as an installation position and an installation angleof a camera on the world coordinate system, a focal distance that is aninternal parameter of a camera, and an effective pixel size of a sensor.In other words, using the above Formulas, the transforming/synthesizingunit 105 converts an image of the actual camera 110 into an image viewedfrom the virtual viewpoint arranged at a certain position.

Subsequently, an example in which the four in-vehicle cameras 110-1 to110-4 are used as described above will be described in an embodiment tobe described below, but the present invention is not limited thereto,and the number of actual cameras 110 may be one. For example, when thefish-eye lens is used as the camera 110, a necessary angle of view maybe covered by one camera depending on the purpose. Thus, the followingembodiment can be applied even when the number of cameras is one.

Here, referring back to FIGS. 2A to 2C, as an example of installation ofa camera in the present embodiment, a state in which the cameras 110-1to 110-4 of the imaging unit 110 in FIG. 1 are mounted on a vehicle 300is illustrated, and in further detail, FIG. 2A is a left side view of avehicle, and FIG. 2B is a plane (top) view. In other words, there is thevehicle 300 on the ground surface 310, the four cameras 110-1, 110-2,110-3, and 110-4 are installed on the front and rear portions and theleft and right portions of the vehicle 300 downwards with apredetermined angle of depression, and Each camera has an angle of viewto have a certain overlapping region between its photographing range anda photographing range of a neighboring camera.

FIG. 2C illustrates imaging ranges of the cameras 110-1, 110-2, 110-3,and 110-4. It is understood in FIG. 2C that the imaging ranges of thecameras 110-1, 110-2, 110-3, and 110-4 are indicated by hatched regions311, 312, 313, and 314, and there are overlapping regions 321, 322, 323,and 324 therebetween. Hereinafter, the present embodiment will bedescribed based on the camera installation condition illustrated inFIGS. 2A and 2B.

A problem in a converted image when the virtual projection plane and thevirtual viewpoint are set without consideration of distortion of anobject will be described with reference to FIGS. 4A and 4B and FIG. 5.

FIGS. 4A and 4B are layouts illustrating an example of the virtualprojection plane and the virtual viewpoint that are set around thevehicle in the present embodiment. Unlike FIGS. 2A and 2B, there is, forexample, an object of interest 401 that is a 3D object such as astanding tree on the left side ahead of the vehicle 300. A plane 412that has the same height as the ground surface 310 and is parallel tothe ground surface is defined, a virtual projection plane configuredwith a virtual spherical surface 411 is arranged at a position fartherthan the object of interest 401, and the plane 412 and the sphericalsurface 411 intersect along a circle 413. In other words, a virtualviewpoint is set at an installation position and an installation angleat which a vehicle body and an area in front of the vehicle are vieweddownwards behind the vehicle 300 as illustrated in FIG. 4A. Thespherical surface 411 need not be exactly a spherical body and may be arotating body of an arbitrary curved line that is designed so that avideo projected onto the virtual projection plane is naturally viewed orany other curved surface.

FIG. 5 illustrates a converted image when a state in which a video ofthe camera 110-1 is projected onto the virtual projection planes 411 and412 set in FIGS. 4A and 4B (hereinafter, referred to as a “virtualprojection state”) is viewed from a virtual viewpoint 414 set in FIGS.4A and 4B. An example of converting the object of interest 401 shown onthe left side ahead of the vehicle 300 into an image photographed by thecamera 110-1 will be described below, but the position of the object ofinterest 401 may be any position around the vehicle 300, and aphotographing camera is not limited to the camera 110-1 and may be anyone of the other cameras 110-2, 110-3, and 110-4.

The video photographed by the camera 110-1 is projected onto thespherical surface 411 serving as the virtual projection plane and theplane 412 as illustrated in FIGS. 4A and 4B. In other words, the objectof interest 401 is projected onto a crossing point 415 of an extensionof a straight line connecting the camera 110-1 with the object ofinterest 401 and the virtual projection plane.

FIG. 5 illustrates an image viewed from the virtual viewpoint, and theimage includes the object of interest 401, a region projected onto thevirtual projection plane 411 serving as the spherical surface, a regionprojected onto the virtual projection plane 412 serving as the plane,and a boundary 413 between the spherical surface and the plane. In theimage viewed from the virtual viewpoint, the ground surface 310 is atthe same level as the planar virtual projection plane 412 because it isat the same position as the planar projection plane, and a non-distortedimage with a sense of perspective can be formed on the region 412, butthe object of interest 401 that is the 3D object such as the standingtree is projected as a surface in a part of the planar projection plane412 and thus is viewed as a distorted image with no a sense ofperspective.

This the same for the region that is the virtual projection plane 411 ofthe spherical surface, and since the object of interest 401 is projectedonto the spherical projection plane 411 that differs in a position (thatis, is deviated), an image viewed from the virtual viewpoint 413 isviewed as a distorted image that collapses obliquely.

The present invention was made to solve the above-described distortion,and in the present embodiment, generation of a converted image that isless distorted will be described with reference to FIGS. 6A and B andFIG. 7.

FIGS. 6A and 6B are layouts illustrating an example of a virtualprojection plane and a virtual viewpoint set around a vehicle 300 in thepresent embodiment, and similarly to FIGS. 3A and 3B, there is an objectof interest 401 that is a 3D object (a standing tree) on the left sideahead of the vehicle 300. In this setting, a virtual sphericalprojection plane 411 is arranged at a position closer to the object ofinterest 401 than that in FIGS. 4A and 4B. A virtual viewpoint 414 isset at a position lower than that in FIGS. 4A and 4B at an installationposition and an installation angle at which the vehicle body of thevehicle 300 and an area in front of the vehicle 300 are viewed downwardsfrom the right rear side.

FIG. 7 illustrates a converted image viewed from the virtual viewpoint414 set in FIGS. 6A and 6B when the video of the camera 110-1 isprojected onto the virtual projection planes 411 and 412 set in FIGS. 6Aand 6B. In this case, as is clear from FIG. 7, in the image of theobject of interest 401, an area projected onto the planar projectionplane 412 is small, a difference in a position (a position deviation)between the spherical projection plane 411 and the object of interest401 is small, and thus distortion on the video is small.

The virtual viewpoint 414 is set so that an angle difference between aline connecting the actual camera 110-1 with a crossing point 415, thatis, a point 415 at which an extension of a line connecting the actualcamera 110-1 with the object of interest 401 crosses the virtualprojection plane 411 and a lie connecting the crossing point 415 withthe virtual viewpoint 414 is a predetermined value or less.

As a result, as is clear from FIG. 7, it is understood that compared tothe converted image illustrated in FIG. 5, distortion in which a senseof perspective of the image 401 of the object of interest is lost isreduced, and distortion in which an object collapses obliquely is lessdisplayed.

As illustrated in FIG. 4A to FIG. 7, a sense of distance around thevehicle 300 and a way in which the object of interest 401 looks greatlychange according to a setting of the virtual projection plane 411 andthe virtual viewpoint 414. In other words, in order to cause the imageof the object of interest 401 such as the standing tree to be viewed tobe less distorted, it is necessary to make the 3D surface of the virtualprojection plane 411 close to the object of interest 401, and it isnecessary to perform a setting so that an angle difference between theline connecting the virtual viewpoint 414 with the object of interest401 and the straight line connecting the actual camera 110-1 with theobject of interest 401 is reduced.

Next, an example of a method of deciding the virtual viewpoint 414 inFIG. 6B from the point of view described above will be described withreference to FIG. 8.

As illustrated in FIG. 8, first, a vector 801 from a point on the objectof interest 401 to the actual camera 110-1 is obtained, and the virtualviewpoint 414 is set within a predetermined angle range 802 (an arrow803) with respect to the vector 801. Accordingly, an angle between thestraight line connecting the actual camera 110-1 with the object ofinterest 401 and the line connecting the virtual viewpoint 414 with theobject of interest 401 is smaller than a predetermined angle, and thusdistortion of a synthesized video can be reduced. For example, when anangle (a minor angle) between the line connecting the virtual viewpoint414 with the object of interest 401 in FIG. 6B and the vector 801 inFIG. 8 is indicated by Δθ, and a distance between the object of interest401 and the point 415 is indicated by D, a distance from a crossingpoint of the virtual projection plane 411 and the line connecting thevirtual viewpoint 414 with the object of interest 401 to the point 415,that is, a deviation amount of the object of interest 401 when thesynthesized video is generated by the virtual viewpoint 414 isconsidered. For the sake of simplicity, when it is assumed that thedeviation amount can approximate to D×Δθ, and an allowable deviationamount of the synthesized video is indicated by E, a range of Δθ isΔθ<E/D, and when the virtual viewpoint 414 is set within the range ofΔθ, the distortion of the synthesized video can be suppressed to bewithin the allowable range.

By setting the virtual viewpoint 414 at a position of Δθ at whichdistortion is smaller than one pixel in an image generated by thetransforming/synthesizing unit 105 according to the distance D betweenthe object of interest 401 and the point 415, it is possible tosynthesize an image in which the object of interest 401 is displayedwith distortion equal to that of the synthesized video viewed from thevirtual viewpoint on the vector 801.

A value of the allowable deviation amount E may be set at the time ofshipment from a factor or may be set by the user. When the value of theallowable deviation amount E is stored in the memory 103 or a storageunit (not illustrated) included in the video synthesis device 100, it ispossible to the virtual viewpoint 414 at an appropriate position evenwhen the distance D between the actual camera 110-1 and the object ofinterest 401 is changed.

Further, when the object of interest 401 is within a region in which theangles of view of a plurality of cameras overlap as in the object ofinterest 401 illustrated in FIG. 6B, the virtual projection plane 411and the virtual viewpoint 414 can be decided so that image conversionfrom a video of another camera 110-2 is performed, for example, asillustrated in FIG. 9 and FIG. 10. In this case, similarly to FIGS. 6Aand 6B, it is desirable that the virtual projection plane 411 be set sothat a minimum distance from the surface of the object of interest 401is a predetermined value or less, and the position and the angel of thevirtual viewpoint 414 be set so that an angle difference with the lineconnecting the left camera 110-2 with the object of interest 401 isreduced in the example of FIG. 9, and when the object of interest 401 isin the overlapping region, the position of the virtual viewpoint 414 isobtained based on the line connecting with the camera 110-1 with theobject of interest 401 or the line connecting the camera 110-2 with theobject of interest 401 according to one of the camera 110-1 and thecamera 110-2 that is higher in a resolution with which the object ofinterest 401 is imaged and the number of pixels when the object ofinterest is formed as an image. One which is smaller in a distancebetween the camera and the object of interest may be selected, or onewhich is close in the traveling direction of the vehicle may beselected.

FIG. 7 and FIG. 10 illustrate the examples of the images obtainedconverted based on the images photographed by the front camera 110-1 andthe left camera 110-2, but when the transforming/synthesizing unit 105(see FIG. 1) in the device synthesizes the images of the respectivecameras through the a blending, an image to be employed may bedetermined, for example, under a condition to be described.

For example, in a state in which the vehicle takes a left turn at theintersection at a relatively low speed, a request for widely displayinga video of a left area having a collision possibility in the travelingdirection is inferred, and in this case, the virtual viewpoint 414 isset as illustrated in FIG. 6B, and the leftward direction is widelydisplayed. Alternatively, when the vehicle is traveling forward at arelatively high speed, and a request for widely displaying a video of afront area having a high collision possibility is inferred, the virtualviewpoint 414 is set as illustrated in FIG. 9, and the forward directionis widely displayed.

Next, FIGS. 11A and 11B illustrate joint setting examples under theconditions illustrated in FIG. 6B and FIG. 9. In both cases, a settingin which a portion in which the object of interest 401 is shown deviatesfrom a joint of camera videos is assumed to be performed in a blendingsynthesis of the overlapping region 321 illustrated in FIG. 2C. In theexamples of FIGS. 11A and 11B, straight lines denoted by thick brokenlines 1101 and 1102 indicate joints at the time of synthesis, and eachvideo is delimited using the lines as a boundary. In FIG. 11A, asdescribed above with reference to FIG. 8, in order to reduce thedistortion of the object of interest 401 in the synthesized video, thevirtual viewpoint 414 is installed to have a predetermined angle withrespect to the line connecting the front camera 110-1 with the object ofinterest 401. At this time, since the image photographed by the frontcamera 110-1 is used as the synthesized image of the object of interest401, the broken line 1101 of the delimiter is set on the left camera110-2 side further than the object of interest 401. Similarly, in FIG.11B, the virtual viewpoint 414 is installed to reduce the distortion ofthe synthesized image at the position of the object of interest 401, andthe image photographed by the left camera 110-2 is used as thesynthesized image, and the broken line 1102 of the delimiter is set onthe front camera 110-1 side further than the object of interest. Inother words, a blending position of the overlapping region 321 isdecided based on a relation among the position of the object of interest401, the position of the camera that images the object of interest 401,and the position of the virtual viewpoint 414, and when the position ofthe virtual viewpoint 414 is arranged to be within a predetermined anglewith respect to the straight line connecting the front camera 110-1 withthe object of interest 401, the blending position is set on the leftcamera 110-2 side so that the video of the object of interest 401 imagedby the front camera 110-1 is used, and when the position of the virtualviewpoint 414 is arranged to be within a predetermined angle withrespect to the straight line connecting the left camera 110-2 with theobject of interest 401, the blending position is set on the left camera110-1 side so that the video of the object of interest 401 imaged by thefront camera 110-2 is used. The broken lines 1101 and 1102 set hereinmay be statically set according to an angle of view of the camera 110-1or the camera 110-2 regardless of the position of the object of interest401 and may be set at a position that does not correspond to a full viewof the object of interest as long as the shape or the size of the objectof interest 401 is known. The present invention is not limited thereto,and a value of a may be changed within a certain range stepwise usingthis line as a boundary, or the lines may be an arbitrary curved linerather than a straight line such as the thick broken lines 1101 and1102. For example, when the object of interest 401 is larger than theoverlapping region 321, and the joint is included on the object ofinterest 401 regardless of a setting position of the joint at the timeof blending synthesis, it may be detected, the joint at the time ofblending synthesis may be set on the object of interest 401 asnecessary, and synthesis may be performed to change a value of a withina certain range using the joint as a boundary. Further, when it ispossible to detect the traveling direction of the vehicle via the CAN orthe like, if the vehicle is traveling forwards, for example, in the caseof FIG. 11A, the blending position may be set on the front camera 110-1side, or when the vehicle takes a left turn in the case of FIG. 11B, theblending position may be set on the left camera 110-2 side. It isbecause, for example, when the vehicle is traveling forwards in FIG.11A, the object of interest 401 moves toward the left side behind thevehicle if relatively viewed, and when the blending position is set to1101, it is necessary to switch the camera that images the object ofinterest 401 from the front camera 110-1 to the left camera 110-2. Thus,when the object of interest 401 imaged by the left camera 110-2 isinitially used for synthesis, it is unnecessary to switch the camera,and it is possible to reduce a possibility that the object of interest401 will disappear from the synthesized image.

In the present embodiment, in order to reduce the computationalcomplexity, the example in which the virtual projection plane isindicated by the simple plane and the spherical surface has beendescribed, but the present invention is not limited thereto, and forexample, a statistical clustering process based on a feature quantitymay be performed according to the accuracy of information detected bythe detecting unit 104 illustrated in FIG. 1, and objects including abuilding, a wall, a guardrail, the sky, and the like may be arranged onthe space together with the ground surface (road) 310 in view of thespatial position relation and the 3D shape thereof and synthesized withthe video. As a result, it is possible to form more accurate 3Dprojection of surrounding areas, that is, it is possible to generate anatural video of a 3D shape having less distortion.

Next, FIG. 12 is a flowchart illustrating an example of a processperformed in the video display system according to the presentembodiment, particularly, a process performed by the CPU 102 of thevideo synthesis device 100, that is, an example of an operation ofdeciding a method of synthesizing the shape of the virtual projectionplane, the installation position and the installation angle of thevirtual viewpoint, and the overlapping region according to the object ofinterest, and a description thereof will be described below in detailaccording to step numbers in which S is added to a beginning partthereof.

First, when a process flow starts in step S1200, in S1201 (an imagerecognizing unit), the detecting unit 104 detects or determines theposition of the object of interest, and acquires a relative position toits own vehicle or an appropriate reference point. As informationacquired herein, a position of the object of interest on a plane view(for example, the standing tree 401 in the above example), that is, aposition of the ground surface 310 is necessary, and as described above,information as to whether or not the object of interest is a 3D objectand a detailed 3D shape thereof may be acquired together.

Then, in S1202, a camera that photographs the object is specified basedon the acquired position of the object of interest. The camera isspecified based on the position of the object of interest, theinstallation position, the installation angle, and the angle of view ofthe camera, or the like. When a plurality of cameras that photograph theobject are specified under this condition, the object of interest isdetermined to be in the overlapping region since there are a pluralityof cameras that photograph the object of interest. However, instead ofusing this condition, a simple method may be used, for example, thecamera that photographs the object of interest may be specified based ona result of image recognition, or a photographing camera may be uniquelyspecified based on the relative position of the object of interest tothe vehicle. It is used as the determination condition in thedetermination step S1207 which will be described later whether or not aplurality of cameras photograph the object of interest duplicatedly.

Then, in S1203, a vector of the object of interest and the photographingcamera is obtained. This vector may be a 2D vector in which a height ona plane view is not considered or may be a 3D vector including a heightas well. Further, when this vector is obtained, a point on the object ofinterest may be designated, or a set of a plurality of vectors may beobtained by designating a plurality of points on the object of interest.

Further, in S1204, a virtual viewpoint position is decided based on thevector obtained in S1203 within a range having an angle differencewithin a certain value with the vector as illustrated in FIG. 8. At thistime, the angle range may have individual values in three axes on theworld coordinates. Thereafter, in S1205, the shape of the virtualprojection plane is decided based on a distance between the object ofinterest and the position of its own vehicle or a distance between theobject of interest and the reference point. For example, in the case ofthe virtual projection planes of the plane and the spherical surfaceillustrated in FIG. 4A to FIG. 11B, the center coordinates and theradius of the spherical body are set so that the position difference(the position deviation) between the object of interest and theprojection plane is reduced. As described above, the projection planemay have a complicated shape according to the 3D shape of the object ofinterest.

Then, in S1206, it is determined whether or not the number ofphotographing cameras specified in S1202 is one or two or more, and asubsequent process is switched based on a result thereof.

First, when the “position is determined to be in the overlapping regionof the camera videos” in S1206, in S1207, in addition to the conditionsof the position of the object of interest and the virtual viewpoint andthe virtual projection plane decided in S1204 and S1205, a camera whosevideo is used is selected among a plurality of cameras that photographthe object of interest, and the synthesis method of the overlappingregion is decided.

On the other hand, when the “position is determined not to be in theoverlapping region of the camera videos” in S1206, in S1208, thesynthesis method of the overlapping region is decided according to theconditions of the position of the object of interest and the virtualviewpoint and the virtual projection plane decided in S1204 and S1205.

Thereafter, subsequently to S1207 or S1208, in S1209, the images of therespective cameras are transformed and synthesized based on the virtualprojection plane, the virtual viewpoint, and the synthesis method of theoverlapping region decided until S1208, and a resulting video is outputto the monitor 120 via the encoding unit 106 of FIG. 1.

An example of the synthesis method of the overlapping region accordingto the present embodiment will be more specifically described withreference to FIG. 13, FIG. 14, and FIG. 15.

First, referring to FIG. 13, the overlapping region 321 surrounded bytwo thick solid lines is used as an example, the overlapping region isdescribed using an angle θ, and a blending rate within a range of θ=θmaxfrom θ=0 is set, for example, as illustrated in FIG. 14 or FIG. 15.

In these settings, a characteristic of the blending rate illustrated inFIG. 14 is simple, and thus an operation load is relatively small, butwhen θ is θ_l or θ_h, the blending rate abruptly changes, and thus anuncomfortable feeling may be added at the time of synthesis of a jointof a video.

On the other hand, compared to that of FIG. 14, a characteristic of theblending rate illustrated in FIG. 15 is complicated but consecutive withrespect to a change in θ, and thus a synthesized video having a lessuncomfortable feeling can be expected to be generated. In the presentinvention, the characteristic of the blending rate is not limited to theexamples illustrated in FIG. 14 and FIG. 15. Reference numerals 1401 and1501 in the drawings indicate the blending rates of the imagephotographed by the camera 110-1, and reference numerals 1402 and 1502indicate the blending rates of the image photographed by the camera110-2. Further, in FIG. 14, blending is steadily changed within a rangeof θ of θ_l θ_h, and pixel values of the videos are weighted andaveraged within this range.

As another transformation example, a synthesized video that enables adetected object of interest to be displayed as a natural image isgenerated, but for example, in the system diagram of FIG. 1, a video ina normal state may be displayed on the monitor 120 in addition to animage showing the object of interest. For example, the virtualprojection plane and the virtual viewpoint in FIG. 4B are set as acondition of a steady state, the virtual projection plane and thevirtual viewpoint in FIG. 6B are set as a condition for display theobject of interest 401, and the transforming/synthesizing unit 105generates a video in the steady state and a video for the object ofinterest, synthesizes the videos so that the videos are divided at acenter of a display screen and arranged, and outputs a resulting videoto the monitor 120 via the encoding unit 106. Accordingly, the drivercan check an area around the vehicle in a wide range and understand theobject of interest such as a pedestrian or an obstacle through a videohaving less distortion.

As another transformation example, when a plurality of objects aredetected around the vehicle, the change of the virtual viewpoint may bechanged so that the detected object (referred to as an “object A”) otherthan the object of interest 401 is less distorted. It is because thedistortion of the object of interest 401 is decided according to anangle formed by the virtual viewpoint 414, the object of interest 401,and the camera 110, and the distance between the virtual viewpoint 414and the object of interest 401 can be arbitrarily set. Specifically, ifa straight line connecting the virtual viewpoint 414 with the object ofinterest 401 is indicated by a straight line A, and a straight lineconnecting the object A with a camera that images the object A isindicated by a straight line B, when the virtual viewpoint 414 is set onthe straight line A nearby a point at which the straight line A isclosest to the straight line B, it is possible to synthesize a video inwhich the object of interest 401 is less distorted, and the object A isrelatively less distorted either.

Further, when three or more objects are detected around the vehicle,priorities of all the objects may be decided through a similar techniqueto the technique of selecting one of a plurality of objects as theobject of interest 401, and the position of the virtual viewpoint 414may be decided by the above-described method so that the object ofinterest 401 having the highest priority and the object A having thenext highest priority are less distorted.

Further, when three or more objects are detected around the vehicle, astraight line connecting an object other than the object of interest 401with a camera that images the object may be virtualized for each object,and the virtual viewpoint 414 may be set on the straight line A nearby aposition at which a distance between the straight lines is smallest.

As the position of the virtual viewpoint 414 is set as described above,it is possible to synthesize a video in which an object other than theobject of interest 401 is less distorted either, and thus usability isimproved.

The exemplary embodiment of the present invention has been describedabove in detail, but the present invention is not limited to the aboveembodiment, and as another embodiment, a plurality of encoding unit 1606and a plurality of monitors 1620 may be provided, and a video may beselectively displayed on each monitor, for example, as illustrated inFIG. 16. In this case, the object of interest to be displayed isswitched according to an installation position or a displaycharacteristic of the monitor, and the transforming/synthesizing unit105 sets at least one of the virtual projection plane, the virtualviewpoint, and the synthesis methods of the overlapping region accordingto each monitor, performs transformation and synthesis, and outputs avideo to the monitors 1620. Accordingly, for example, it is possible todisplay an appropriate video more naturally according to a feature of amonitor, for example, it is possible to cause a video in a steady stateto be displayed on a navigation screen and cause a video around avehicle that is not viewed from a driver to be displayed on a head updisplay (HUD).

As another embodiment, as illustrated in FIG. 17, a plurality oflightings 1721 and a plurality of blinking control units 1707 thatcontrol blinking of a plurality of lightings are provided, and it ispossible to notify of the position of the object of interest throughboth a video and blinking of a lighting. As an example, lightingscapable of blinking independently are installed at front, rear, left,and right portions in a vehicle, and for example, when there is adangerous object on the left side behind the vehicle, an area on theleft side behind the vehicle is noticeably displayed on the monitor 120,and the lighting near the left rear portion blinks at the same time.Accordingly, the driver can be informed of the dangerous object moreintuitively.

As another embodiment, as illustrated in FIG. 18, a plurality ofspeakers 1821 and a plurality of rumbling control units 1807 thatcontrol rumbling of a plurality of speakers may be provide, and it ispossible to notify of the position of the object of interest throughboth a video and a sound. As an example, speakers capable of rumblingindependently are installed at front, rear, left, and right portions ina vehicle, and for example, when there is a dangerous object on the leftside behind the vehicle, an area on the left side behind the vehicle isnoticeably displayed on the monitor 120, and a warning sound is outputfrom the speaker near the left rear portion blinks at the same time.Accordingly, the driver can be informed of the dangerous object moreintuitively.

As described above, in an embodiment of the present invention, a videodisplay system performs a video conversion process on a video of acamera mounted on a vehicle and displays a resulting video, and includesa plurality of cameras, a detecting unit that detects an object ofinterest around a vehicle based on information or the like acquiredthrough the plurality of cameras, other sensors, or a network, atransforming/synthesizing unit that transforms and synthesizes thevideos photographed by the plurality of cameras using a shape of avirtual projection plane, a virtual viewpoint, and a synthesis methodwhich are decided according to position information of the object ofinterest detected by the detecting unit, and a display unit thatdisplays the video that is transformed and synthesized by thetransforming/synthesizing unit.

Further, in an embodiment of the present invention, in a video displaysystem that performs a video conversion process on a video of a cameramounted on a vehicle and displays a video that has undergone the videoconversion process, a video obtained by performing transformation andsynthesis on vides photographed by a plurality of cameras through atransforming/synthesizing unit using a shape of a virtual projectionplane, a virtual viewpoint, and a synthesis method which are decidedaccording to position information of an object of interest is set as afirst synthesized video, and a second synthesized video obtained byperforming transformation and synthesis on the videos photographed bythe plurality of cameras in a state in which the virtual viewpoint isnot changed even when the position information of the object of interestis changed is displayed on the display unit or a second display unitseparate from the display unit, separately from the first synthesizedvideo.

Further, in an embodiment of the present invention, a video displaysystem performs a video conversion process on a video of a cameramounted on a vehicle and displays a video that has undergone the videoconversion process, and includes one or more lights arranged in thevehicle and a blinking control unit that controls blinking of thelighting, and a notification of a position information of an object ofinterest detected by a detecting unit is given by a position at whichthe lighting blinks.

Further, in an embodiment of the present invention, a video displaysystem performs a video conversion process on a video of a cameramounted on a vehicle and displays a video that has undergone the videoconversion process, and includes one or more speakers arranged in thevehicle and a rumbling control unit that controls rumbling of thespeaker, and a notification of a position information of an object ofinterest detected by a detecting unit is given by a position at whichthe speakers outputs a sound.

The present invention is not limited to the above embodiment, andvarious transformation examples are included. The above embodiment hasbeen described in detail to facilitate understanding of the presentinvention, and the present invention is not limited to a configurationnecessarily including all the components described above. Further, somecomponents of a certain embodiment may be replaced with components ofanother embodiment. Further, components of another embodiment may beadded to components of a certain embodiment. Furthermore, othercomponents may be added to, deleted from, and replace some components ofeach embodiment.

All or some of the above components, functions, processing units,processing means, or the like may be implemented by hardware such thatthey are designed by, for example, as integrated circuit (IC). The abovecomponents, functions, or the like may be implemented by software byinterpreting and executing a program of implementing the functionsthrough a processor. Information such as a program, a table, or a filefor implementing each function may be stored in a recording apparatussuch as a memory, a hard disk, a solid state drive (SSD) or a recordingmedium such as an IC card, an SD card, or a DVD.

REFERENCE SIGNS LIST

-   100 video synthesis device-   101 decoding unit-   102 central processing unit (CPU)-   103 memory-   104 detecting unit-   105 transforming/synthesizing unit-   106 encoding unit-   107 bus-   110 imaging unit-   110-1 to 110-4 camera-   120 monitor-   300 automobile

1. (canceled)
 2. A video synthesis device that transforms andsynthesizes a plurality of videos input from a plurality of cameras andoutputs a synthesized video to a video display unit, comprising: anobject-of-interest information acquiring unit that acquires positioninformation related to an object of interest; and atransforming/synthesizing unit that transforms and synthesizes theplurality of videos photographed by the plurality of cameras, whereinthe transforming/synthesizing unit transforms and synthesizes a virtualprojection state in which the videos photographed by the plurality ofcameras are projected onto a virtual projection plane into a videoviewed from a virtual viewpoint set according to position informationrelated to the object of interest, and the transforming/synthesizingunit sets the virtual viewpoint at a position at which an angle formedby a straight line connecting the virtual viewpoint with the object ofinterest and a straight line connecting the object of interest with acamera that images the object of interest among the plurality of camerasis smaller than a predetermined angle.
 3. The video synthesis deviceaccording to claim 2, wherein the predetermined angle is decided so thatdistortion of a video of the object of interest in the synthesized imageis minimized according to a distance between the object of interest andthe camera that images the object of interest and a deviation amountbetween a projection position at which the object of interest isprojected onto the virtual projection plane and a position of the objectof interest.
 4. (canceled)
 5. The video synthesis device according toclaim 2, wherein the virtual projection plane is configured with a planethat has the same height as a ground surface and is parallel to theground surface and a spherical surface arranged at a position fartherthan the object of interest, and the transforming/synthesizing unit setsat least one of a center position of the spherical surface and a radiusof the spherical surface so that the spherical surface of the virtualprojection plane is arranged within a predetermined distance from aposition of the object of interest when viewed from the camera thatphotographs the object of interest.
 6. The video synthesis deviceaccording to claim 2, wherein the transforming/synthesizing unitsynthesizes the videos by blending videos obtained by transforming avirtual projection state in which the plurality of videos are switchedat a blending position decided according to the position informationrelated to the object of interest and projected onto the virtualprojection plane into the videos viewed from the virtual viewpoint. 7.The video synthesis device according to claim 6, wherein thetransforming/synthesizing unit synthesizes, when the object of interestexists in a region that is duplicatedly photographed by two neighboringcameras, the videos by blending videos obtained by setting the blendingposition so that the object of interest does not overlap the blendingposition and transforming the videos photographed by the two cameras. 8.The video synthesis device according to claim 2, wherein the object ofinterest is an object that is inferred based on information related to aspeed of the vehicle with which the video synthesis device is equipped,information related to a steering angle, and information related to atraveling direction and exists in a direction having a possibility of acollision with the vehicle.
 9. (canceled)
 10. A video synthesis methodof transforming and synthesizing videos input from a plurality ofcameras and outputting a synthesized video to a video display unit,comprising: an object-of-interest information acquisition step ofacquiring position information related to an object of interest; and atransformation/synthesis step of transforming and synthesizing theplurality of videos photographed by the plurality of cameras, wherein inthe transformation/synthesis step, a virtual projection state in whichthe videos photographed by the plurality of cameras are projected onto avirtual projection plane is transformed and synthesized into a videoviewed from a virtual viewpoint set according to the positioninformation related to the object of interest, and in thetransformation/synthesis step, the virtual viewpoint is set at aposition at which an angle formed by a straight line connecting thevirtual viewpoint with the object of interest and a straight lineconnecting the object of interest with a camera that images the objectof interest among the plurality of cameras is smaller than apredetermined angle.
 11. The video synthesis method according to claim10, wherein the predetermined angle is decided so that distortion of avideo of the object of interest in the synthesized image is minimizedaccording to a distance between the object of interest and the camerathat images the object of interest and a deviation amount between aprojection position at which the object of interest is projected ontothe virtual projection plane and a position of the object of interest.12. (canceled)
 13. The video synthesis method according to claim 10,wherein the virtual projection plane is configured with a plane that hasthe same height as a ground surface and is parallel to the groundsurface and a spherical surface arranged at a position farther than theobject of interest, and in the transformation/synthesis step, at leastone of a center position of the spherical surface and a radius of thespherical surface is set so that the spherical of the virtual projectionplane is arranged within a predetermined distance from a position of theobject of interest when viewed from the camera that photographs theobject of interest.
 14. The video synthesis method according to claim10, wherein in the transformation/synthesis step, the videos aresynthesized by blending videos obtained by transforming a virtualprojection state in which the plurality of videos are switched at ablending position decided according to the position information relatedto the object of interest and projected onto the virtual projectionplane into the videos viewed from the virtual viewpoint.
 15. (canceled)